OHS must not contain any robots.txt files.

Overview

Finding ID	Version	Rule ID	IA Controls	Severity
V-221464	OH12-1X-000227	SV-221464r879887_rule		Medium

Description
Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web-site content. In turn, these search engines make the content they obtain and catalog available to any public web user. To request that a well behaved search engine not crawl and catalog a server, the web server may contain a file called robots.txt for each web site hosted. This file contains directories and files that the web server SA desires not be crawled or cataloged, but this file can also be used, by an attacker or poorly coded search engine, as a directory and file index to a site. This information may be used to reduce an attacker’s time searching and traversing the web site to find files that might be relevant. If information on hosted web sites needs to be protected from search engines and public view, other methods must be used.

Description

Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web-site content. In turn, these search engines make the content they obtain and catalog available to any public web user. To request that a well behaved search engine not crawl and catalog a server, the web server may contain a file called robots.txt for each web site hosted. This file contains directories and files that the web server SA desires not be crawled or cataloged, but this file can also be used, by an attacker or poorly coded search engine, as a directory and file index to a site. This information may be used to reduce an attacker’s time searching and traversing the web site to find files that might be relevant. If information on hosted web sites needs to be protected from search engines and public view, other methods must be used.

STIG	Date
Oracle HTTP Server 12.1.3 Security Technical Implementation Guide	2022-12-09

Details

Check Text ( C-23179r415075_chk )
1. Open $DOMAIN_HOME/config/fmwconfig/components/OHS//httpd.conf and every .conf file (e.g., ssl.conf) included in it with an editor that contains a "" directive. 2. Search for the "DocumentRoot" directive at the OHS server and virtual host configuration scopes. 3. If the directive value specifies a directory containing a robots.txt file, this is a finding.

Fix Text (F-23168r415076_fix)
1. Open $DOMAIN_HOME/config/fmwconfig/components/OHS//httpd.conf and every .conf file (e.g., ssl.conf) included in it with an editor that contains a "" directive. 2. Search for the "DocumentRoot" directive at the OHS server and virtual host configuration scopes. 3. Remove any robots.txt files from the directories specified in the "DocumentRoot" directives.